AI Disclosure: This code was written in part by an AI agent.:

AI Disclosure: This code was written in part by an AI agent.:

…m fix

The example codecs restore objects from a process-local token registry and
never read the `TaskContext` their FFI decode callbacks are handed, so
which session that context belongs to was untestable. The token path
ignores the registry entirely, which is why an empty
`SessionContext::new()` has served as the exported provider without
anyone noticing.

Both codecs now accept `require_udf_on_decode`. When set, every decode
call resolves that scalar function out of the task context it was given
and fails with the session id if it is absent, which makes the answer
observable. Each codec registers a marker function on the context it
exports, so a name owned by the codec's library and a name owned by the
host can be told apart.

Four tests use it. The two library-local cases pass: a foreign codec
resolves against the session its own library supplied. The two
host-registered cases are `xfail(strict=True)`, because a function
registered on the host with `register_udf` is not visible to a foreign
codec's decode callback at all. A fifth pins the current error so the
failure mode stays legible. Strict xfail means the pair will announce
itself if the upstream design changes.

Document the rule this establishes, and correct the surrounding section:
`with_query_planner` rebuilds a foreign planner against the session that
will run the query, so the provider a planner library supplies is
replaced on that path. Codecs installed through
`with_logical_extension_codec` and `with_physical_extension_codec` keep
the provider their own library exported, which is the case these tests
exercise.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`FFI_QueryPlanner::new` and `FFI_{Logical,Physical}ExtensionCodec::new`
ask an extension library for a `TaskContextProvider`, and a planner for
two codecs on top of that. A library has none of those. Both examples
answered with `Arc::new(SessionContext::new())`, an empty session that
resolves nothing, held weakly by `FFI_TaskContextProvider` and therefore
also a lifetime hazard.

The table provider protocol already solved this: the host calls
`__datafusion_table_provider__(session)` and the library takes what it
needs off the session. Do the same for the other three getters.
`__datafusion_query_planner__`, `__datafusion_logical_extension_codec__`,
and `__datafusion_physical_extension_codec__` now receive the
`SessionContext` they are being installed on. A codec takes the task
context provider from it; a planner takes both codecs and uses
`new_with_ffi_codecs`, which needs no provider at all. Neither example
constructs a `SessionContext` any more.

Decode callbacks consequently resolve against the session running the
query. The two `xfail(strict=True)` tests from the previous commit now
pass unmodified: a scalar function registered on the host with
`register_udf` is visible inside a decode callback executing in another
library, for both the logical and physical codec. A negative control
keeps the check honest, and a further test covers a function registered
after the codec was installed, since the provider is a live handle rather
than a snapshot.

`PySessionContext` gains an `ancestors` list. A foreign codec is built
against the session current at the time it is installed and holds it
weakly, so installing a foreign planner afterwards — which forks — would
strand the codec once the Python name is rebound. The keep-alive lives on
`PySessionContext` rather than on the codec because nothing reachable
from a `SessionContext` reaches a `PySessionContext`, so it cannot close
a cycle. What it does not paper over is the fork itself: a function
registered after the fork is not visible to a codec bound to the session
before it, which is the existing derived-context caveat seen from the
codec's side, and is covered by a test.

`SessionContext` accepts and ignores the argument on all three getters,
so a session satisfies the same protocol a library implements and
`ctx.__datafusion_query_planner__()` keeps working for the delegation
pattern. Calling a stale getter that takes no session now reports an
incompatible-library error naming the method, matching what
`table_provider_from_pycapsule` does.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The session-passing rule was already settled for four getters and
documented in the 52.0.0 upgrade guide, but nothing pointed an agent or a
new contributor at it before they wrote a fifth. Write it down where it
will be found.

Add the 55.0.0 upgrade guide entry this branch owes. Changing
`__datafusion_logical_extension_codec__` and
`__datafusion_physical_extension_codec__` to take a session breaks every
extension library implementing them, so it needs before/after Rust in the
same shape as the 52.0.0 entry.

Correct `user-guide/io/table_provider.md`. It still showed the pre-52.0.0
signature with no session and a `PyCapsule::new_bound` call, so the one
page a reader is most likely to find contradicted the convention.

Add `.ai/skills/ffi-capsule-protocol/`. Its description is written as a
trigger rather than a task, because the existing skills are all things to
run on request and a convention read as one would be skipped. It leads
with enumerating the family, which is the step that makes the rest
unnecessary.

Point `CLAUDE.md` at it, since that file loads unconditionally and a skill
only helps once someone goes looking. Also note that `docs/temp/` is
gitignored build output that `grep -r` surfaces with stale copies, and
require an upgrade guide section alongside the `api change` label, so a
breaking change forces a visit to the file that records the conventions.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Installing a foreign query planner writes to `SessionState`, and
`with_query_planner` must not modify its receiver, so it forks. A foreign
codec holds an `FFI_TaskContextProvider` pointing at the session it was
installed on, and until now the fork could not move it: passing a new
provider to `FFI_LogicalExtensionCodec::new` was silently discarded
whenever the codec was already foreign. The fork rebound only its own
outer wrapper, so decode callbacks in the extension library kept
answering from the pre-fork registry, and the pre-fork session had to be
retained or the weakly held provider dangled.

apache/datafusion#24722 fixes the discard; those constructors now adopt
the provider on the already-foreign path. Repoint the patch at the branch
carrying it and rebind both codecs onto the fork.

Verified the branch carries everything already pinned rather than trusting
the commit graph, which reports the two as diverged: across 3811 files the
only differences are the four constructors from the fix, and
`datafusion/ffi/src/session/mod.rs` is byte-identical, so the
`create_physical_plan` codec fix arrives as its branch-55 backport.

`ancestors` and its helpers are deleted. They existed only to keep the
pre-fork session alive for a codec that could not be moved off it, and a
codec bound to the running session needs no such anchor.

Three tests, replacing two that were weaker than they looked. One
registers a function on the fork after the codec was installed on its
parent and resolves it, which is the direct evidence the rebind happened;
it failed before this change. One installs a planner twice and asserts the
first context still cannot resolve a function registered only on the
second, covering the clone-before-adopt half — a rebind that mutated the
shared handle would pass the first test and fail this one. The third keeps
the live-handle case. The test it replaces required a name registered
nowhere, so it passed for the same reason as the negative control and
never exercised a fork at all.

Note the version floor in `Cargo.toml` rather than raising it now: the
patched branch still reports 55.0.0, so the requirement can only move to
55.1.0 when the patch section is removed. Building against 55.0.0 without
the patch would compile and silently skip the rebind.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Same drift just fixed in derived_parts, but on a path that never forks.
add_physical_optimizer_rule rebuilds SessionState through
SessionStateBuilder::new_from_existing and writes it straight back into
the caller's own session, so the fresh id build() mints replaces the one
SessionContext had already cached at construction. The session the user
is holding then reports one id from session_id() and a different one
from every TaskContext it hands out, with no derivation to explain it.

Reproduced against a foreign codec, reading the id back out of a decode
callback: identical setup differing only by an add_physical_optimizer_rule
call went from MATCH to DRIFT, and back to MATCH with the id threaded
through the builder.

This is the last new_from_existing call site in the crate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The two session id fixes had no regression guard. A Python-level
assertion cannot provide one: session_id() reads a copy SessionContext
caches at construction, which stayed correct through both bugs. The id
that actually moved was the one inside the TaskContext handed to a
foreign codec's decode callback, which nothing exposed.

Give the example codecs a TaskContextProbe that records it. This
replaces the bare AtomicUsize the require_udf_on_decode support used, so
the counter and the session id are recorded together, and the id is
recorded on every decode rather than only when a function was requested.

Three tests, all against the codec-side id rather than session_id():
a fork agrees with its codecs, add_physical_optimizer_rule does not move
the id, and a two-deep fork chain leaves both halves on the parent's id.

Confirmed non-vacuous: with both fixes reverted all three fail and the
other 17 tests pass; with only the derived_parts fix restored, exactly
the add_physical_optimizer_rule test still fails.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

call_capsule_getter rewrote every TypeError from a capsule getter into
"Incompatible libraries ... Upgrade the library providing this object",
and dropped the original. Only an arity mismatch means the library is
out of date. An extension author whose own getter raised a TypeError --
a bad cast, a wrong argument to something it called -- was told the
error was a version problem and lost the error that would have located
it.

The two are distinguishable without guessing at message text: an arity
mismatch is raised by the call machinery before the getter's frame
exists, so no frame unwinds and no traceback is attached, while an error
from the body carries one. Verified to hold for both pure-Python and
pyo3-compiled getters, which is the case that matters here since
extension libraries are compiled.

Also chains the original as __cause__ on the paths that do report an
upgrade, so the arity error stays readable.

Tests cover all three outcomes. Confirmed non-vacuous: dropping the
traceback check fails only the inside-the-getter test, dropping
set_cause fails only the upgrade test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The from_pycapsule! macros call the getter with no arguments. That is
correct for __datafusion_physical_optimizer_rule__ and
__datafusion_task_context_provider__, which take no session, but
__datafusion_physical_extension_codec__ now takes the session it is
being installed on, so this helper was the one member of the family left
speaking the old protocol.

Nothing in the tree called it, but datafusion-python-util is published
by `cargo publish --workspace`, so it was still reachable. Against an
updated codec it raised a bare TypeError, bypassing the ImportError that
names the method. Against an outdated one it succeeded and produced a
codec resolving names against the wrong session -- the silent failure
the rest of this work exists to prevent.

Removing it is a breaking change to that crate, but the crate already
breaks this release: ffi_logical_codec_from_pycapsule gained its session
parameter. A compile error pointing at the replacement beats a helper
that quietly binds to nothing.

Callers move to ffi_physical_codec_from_pycapsule, which passes the
session, plus (&ffi).into() where an Arc<dyn PhysicalExtensionCodec> is
wanted -- what crates/core already does.

Documents both helper changes in the 55.0.0 upgrade guide, which until
now covered only the __datafusion_*__ method signatures and not the Rust
helpers the same authors call.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Only ffi_query_planner_from_pycapsule validated the version a capsule
reported. The codec and table provider importers dereference a foreign
struct through the same `unsafe { data.as_ref() }` and were happy to
accept one built against a different DataFusion.

Extracts the planner's inline check into check_ffi_version and applies
it to the logical codec, physical codec, and table provider importers as
well. The helper is pub so extension libraries writing their own
importers can use it.

Two things the symmetry cannot reach, both now documented where someone
would look:

FFI_TaskContextProvider, FFI_TableProviderFactory, and
FFI_ExtensionOptions carry no version field, so their importers cannot
check. The from_pycapsule!/try_from_pycapsule! macros are #[macro_export]
and generic over the FFI type, so requiring a version field there would
break downstream users holding one of those three; they stay unchecked
and their doc comment now says to call check_ffi_version directly.

This is a diagnostic, not a soundness guarantee, and the helper says so:
`version` is not the first field on any of these structs, so reading it
already assumes the local layout. It turns the realistic failure -- a
library compiled against a different DataFusion -- into a clear error
instead of undefined behaviour on first use, which is what
datafusion_ffi::version is documented to be for.

Verified all four sites are wired by inverting the comparison and
confirming each one fires from the test suites.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Exact equality is only right while datafusion_ffi::version tracks the
crate's semver major, which it does today, so the number moves on every
major release whether or not the ABI changed. If a version span later
becomes compatible, a maintainer needs to know that this one body holds
the whole policy -- callers pass a value and no decision -- and that
relaxing it at a call site would reintroduce the split the helper was
added to remove.

Also records the likelier resolution: if the ABI is stable but version
still follows the crate major, upstream's compatibility marker is wrong
for every consumer, so the fix belongs there rather than in a local
range policy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The table provider and table function importers each carried their own
copy of the TypeError-to-ImportError mapping, predating
call_capsule_getter and never folded into it. Both therefore missed the
correction it since received: they rewrote a TypeError raised inside a
correctly-signed getter into "upgrade your library", and discarded the
original.

Three copies of one mapping, two of them stale, is the reason to have
one. Both now call the shared helper, so they pick up the traceback
discrimination and the __cause__ chain, and any later correction reaches
all three by construction.

Their messages named DataFusion 52.0.0. The shared message names the
method that refused the argument instead, which points at the specific
hook rather than a release, and the upgrade guide carries the version
detail. call_capsule_getter is now pub, with a doc comment saying to use
it rather than calling getattr directly.

Tests cover both outcomes on both paths. Verified against the previous
build that they are non-vacuous: before this change the raises-inside
case produced the same misleading ImportError as the old-signature case.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The before and after snippets pass the task context provider
differently, by reference in one and by value in the other, with nothing
saying why. Read as a diff it looks like a typo in one of them, and a
reader correcting it would be puzzled when both versions compile.

Both are valid: the parameter is impl Into<FFI_TaskContextProvider>,
which is satisfied by &Arc<dyn TaskContextProvider> and by
FFI_TaskContextProvider itself, and the latter is what
ffi_task_context_provider_from_pycapsule returns. The argument changes
because the provider now comes from the session instead of a field,
which is the point of the migration.

The contributor guide shows only the post-migration form, so it needs no
equivalent note.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The README already describes these as one-shot registries that consume
each token during decoding, but the source comments did not, and the
source is what someone reuses the pattern from. The existing comment
warned that the registry is process-local without saying that a decode
removes its entry, which is the constraint most likely to bite.

Documents both consequences on the registry accessors, where the
mechanism lives, with a pointer from each struct doc:

- Decode consumes the token, so the same encoded bytes cannot be decoded
  twice. Fine here because every plan is encoded immediately before the
  one decode that consumes it, but it rules out replaying a stored plan,
  retrying a decode, or fanning one plan out to several readers.
- An encode that never reaches a decoder leaks for the life of the
  process. Normal operation does not: encode and decode counts balance
  exactly across repeated queries, which is what makes remove-on-decode
  the right trade here rather than a leak on every call.

Comments only.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

MyQueryPlanner::new imported its fallback immediately, with no session
to pass, so the fallback's getter was called with no arguments. That
works for a SessionContext, whose getter takes the session optionally,
and for a raw capsule, which has no getter at all. It fails for another
foreign planner, which implements the same protocol this type does and
requires the argument -- and layering on another planner is the case a
distributed engine actually needs. The docstring claimed fallback "takes
anything exporting __datafusion_query_planner__", which was not true.

Holds the Python object instead and imports it in
__datafusion_query_planner__, where the session is in hand and can be
forwarded. All three fallback kinds now work.

Deferring also removes a footgun rather than adding one. Passing a
SessionContext now delegates to whichever planner it holds at install
time, and since with_query_planner calls the getter before installing,
the context still reports its previous planner, so wrapping a context in
a planner installed on that same context does not recurse.

Arc<Py<PyAny>> rather than Py<PyAny> because pyo3 0.29 gates Py: Clone
behind the py-clone feature, and this type derives Clone. Matches how
PythonTableFunctionCallable holds its callable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

foreign_session, foreign_provider, and foreign_plan were written with
store, so each one described only the most recent plan. Their accessors
are named foreign_*_observed, which asks whether the thing was ever
seen, and the tests assert them after running more than one query. The
existing tests passed by luck.

Reproduced: after scanning a foreign provider and then running
SELECT 1, foreign_provider_observed goes from True back to False.

Writes them with fetch_or so a later plan cannot retract what an earlier
one observed. plan_calls already accumulated, used_fallback only ever
stores true so it was already cumulative, and last_max_rows is
deliberately last-wins as its name says. Documents that split on the
struct, since it is the kind of thing that gets "tidied" back.

Confirmed non-vacuous: with store restored, exactly the new test fails
and the other 22 pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`with_query_planner` derived a new `SessionContext` to install a planner,
on the grounds that the receiver must not be modified. That mints a fresh
`Arc<SessionContext>` allocation, and every FFI handle in play is bound to
an allocation rather than to the logical session: `FFI_TaskContextProvider`
holds its provider weakly, and a registered catalog provider upgrades that
handle on every `supports_filters_pushdown` and every `scan`.

So the natural `ctx = ctx.with_query_planner(planner)` dropped the session
a foreign catalog had been registered on, and the next query failed with
`TaskContextProvider went out of scope over FFI boundary`. Reproduced with
a `MyCatalogProvider` registered before the install and a `WHERE` clause to
force pushdown during logical optimization.

Rebinding cannot cover this. It reaches the codecs `PySessionContext` holds
in its own fields; a codec embedded in a registered `FFI_CatalogProvider` —
and in every `FFI_SchemaProvider` and `FFI_TableProvider` minted from it —
has no Python-side handle. Nor can a codec retain the session that built
it: codecs are routinely handed to a provider that is registered straight
back into that session, closing `SessionContext -> catalog -> FFI provider
-> FFI codec -> SessionContext`.

Install in place instead, writing `SessionState` back through `state_ref()`
exactly as `add_physical_optimizer_rule` already did. A session keeps one
`Arc<SessionContext>` for life, so no handle is ever orphaned and the bug
cannot occur. This deletes the fork and everything that existed to repair
it: `ancestors`, `rebound_{logical,physical}_codec`, `exported_session` on
both codecs, and the `exported_ffi_*` builders.

The query planner lives in `SessionState`, so it belongs to the session
rather than to a handle on it. `with_query_planner(planner) ->
SessionContext` therefore becomes `set_query_planner(planner) -> None`,
matching `add_physical_optimizer_rule`.

The 55.1.0 pin may no longer be needed — its stated reason in Cargo.toml is
the rebinding this removes — but that is left alone pending a check of the
rest of the PR.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The test registered a foreign catalog provider and no codecs, so it could
never finish: once the planner install stopped orphaning the provider, the
query got past filter pushdown and then failed at plan serialization with
`LogicalExtensionCodec is not provided`. That is the same unrelated failure
`test_query_planner_requires_provider_codec` already covers, and it would
mask a dangling handle rather than expose one.

Install both provider codecs, and fold the codec-install-after-planner case
in as a parameter rather than a near-duplicate test. Both orderings write
`SessionState` — one installs the planner, the other rebuilds it against a
new codec — so both exercise the path that must not replace the session's
`Arc<SessionContext>`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`with_logical_extension_codec` and `with_physical_extension_codec` built
the replacement wrapper with `Python{Logical,Physical}Codec::new`, whose
constructor defaults `python_udf_inlining` to true. Installing a codec on
a context that had opted out therefore turned inlining back on without
saying so:

    ctx = SessionContext().with_python_udf_inlining(enabled=False)
    ctx = ctx.with_logical_extension_codec(codec)   # inlining silently back on

That matters beyond a stale flag. Inlining is what embeds a cloudpickled
callable in the wire format, and it is opt-out precisely because that is
not portable across interpreters and not something every deployment wants
to ship. A codec install is not a request to change it.

Carry the receiver's setting across instead. Both new tests fail on the
prior build with `DFPYUDF` reappearing in the blob, and the paired
`..._preserves_inlining_when_enabled` case pins the default-on direction so
the fix cannot degenerate into hard-coding it off.

The physical case is covered in `test_plans.py` rather than alongside the
logical one: `Expr.to_bytes` only routes through the logical codec, so an
assertion there would pass with the physical bug still present. It takes an
`ExecutionPlan` to observe.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`call_capsule_getter` claimed every capsule getter went through it, so the
mapping from a refused argument to a diagnosable error would live in one
place. Seven sites still called `getattr(...).call0()` or `.call1(...)`
directly, so the claim was false and four of them — catalog provider,
schema provider, catalog provider list, table provider factory — still
handed an out-of-date extension library a bare `TypeError`.

Those four take the host's logical extension codec rather than the session,
which is why they could not simply be passed through as they stood: the
diagnostic would have told a catalog author their method "must accept the
SessionContext", pointing them at the wrong parameter. Carry the argument
and its description together in a `CapsuleGetterArg` so one diagnostic can
serve getters that take a session, getters that take a codec, and getters
that take nothing. `Option<&Bound<PyAny>>` still converts into it, so the
documented `*_from_pycapsule` helper signatures are unchanged.

The three zero-argument getters (scalar, aggregate, window UDF) route
through as well. Nothing can be refused there, but the rule is easier to
follow with no exceptions to remember.

Also add `validate_pycapsule` to `table_provider_from_pycapsule` and
`ffi_logical_codec_from_pycapsule`, the two extraction sites that lacked
it. This is not redundant with `pointer_checked`, despite appearances:
`pointer_checked` bottoms out in CPython's `PyCapsule_GetPointer`, whose
error is the fixed string `PyCapsule_GetPointer called with incorrect
name` and names neither the expected capsule nor the one received. Say so
in a doc comment so it does not get "simplified" away later.

Drop the unused `datafusion-proto` dependency from the query planner
example while here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The Cargo.toml comment justified the pre-release pin by "the FFI codec
rebinding in `PySessionContext::derived_parts`", a symbol that no longer
exists. That rebinding was not removed, it moved into
`set_session_query_planner`, which every `with_*` method calls. It depends on
`FFI_QueryPlanner::new_with_ffi_codecs` unwrapping a `ForeignQueryPlanner` and
replacing its codecs, a swap that is a silent no-op before 55.1.0
(apache/datafusion#24722). So the pin is still required, not droppable.

The `with_*` methods rebuild the installed planner on the *shared* session, so
the rebind takes effect even when the returned context is discarded.
`with_python_udf_inlining` additionally claimed "the original session is
unchanged", which the rebuild contradicts; it is the context's own codec
settings that are unchanged.

Also:
- Note that the arity-vs-body TypeError split in `call_capsule_getter` holds
  only because the call originates in Rust. A Python-level shim between the
  host and the getter would supply a traceback and silently disable it.
- Document the new FFI major-version gate in the 55.0.0 upgrade guide. Table
  providers previously performed no such check, so a mismatched extension
  library that used to load now raises ImportError.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…nings

`test_a_discarded_derived_context_still_rebinds_the_planner` covers the
surprising half of the shared-session rebuild: a codec installed through a
context that is then thrown away still binds to the session's planner. A fresh
codec instance makes it observable, since the planner encodes the outbound
logical plan with whichever codec it holds. Verified non-vacuous -- a codec
built but never installed reports zero encode calls.

The query planner example had no conftest, so it ran without the autouse
fail-on-log-warning handler the provider example uses, despite calling
`pyo3_log::init()` for the same reason. Copied verbatim; the suite passes
under it with no allowlist needed.

Two table provider tests called the deprecated `register_table_provider`,
which is a one-line forwarder to `register_table`, so they reached the same
capsule path while emitting DeprecationWarning.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

It locked nothing -- a single editable entry for the crate itself and no
dependencies. Nothing consumed it either: CI runs both example suites with
`uv run --no-project`, and the older datafusion-ffi-example has no lock file
at all. The codespell skip list in pyproject.toml matches on a bare `uv.lock`
glob, so it still covers the repository root lock.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>